Secondary escalation policy if Alert triggers X times during a window

#1

I may have a bit of a unique use case. Some stakeholders have asked if they could be alerted if a specific alert is triggered X times (for instance 2) over a certain time window (lets say 1 hour). Such that at that point they would be woken up.

Manually, we can add a responder if the person receiving the alert realizes that this fired multiple times in that window. However, I am seeking some type of automated mechanism to do this.

#3

Hello!

One method that we offer this with is setting threshold alerts. This would essentially suppress the alerts coming in for the specific rule and only create an incident should 2 alerts were created in one hour.

The problem with this and your use-case is that:

  1. You would not be able to use this with stakeholder accounts so they would require being transferred to a team responder role.
  2. This would also prevent the actual on-call users from receiving the first alert.
    • A workaround for this could entail duplicating the two rules so that one is routed to a service with actual responders and another is routed to the users who don’t require notifications unless related to a high-impact event.

If you are adding the stakeholder as a responder, then this situation could work for you!

#4

I think the new Analytics Module (and possibly the Visibility Module) might give you the insights your stakeholder is looking for.

What’s the reason your stakeholders want to capture this repeating issues here? Do you have consistent first responders who are already performing this manually (ie a NOC)? Are these stakeholders holding full PD accounts or Stakeholder accounts (in terms of adding them to Incidents, etc).

If you have alerts which are occurring frequently, but are being resolved by action (or inaction), you’d probably be better off with a separate (manual) Incident that won’t be automatically resolved on the next resolution of the alert. What action will the stakeholders take once they’ve been woken up?

Sounds like an interesting question you’re fielding there.